On Comparison of Adaptive Regularization Methods

Modeling with exible models, such as neural networks, requires careful control of the model complexity and generalization ability of the resulting model which nds expression in the ubiquitous bias-variance dilemma [4]. Regularization is a tool for optimizing the model structure reducing variance at the expense of introducing extra bias. The overall objective of adaptive regularization is to tune the amount of regularization ensuring minimal generalization error. Regularization is an supplement to direct model selection techniques like step-wise selection and one would prefer a hybrid scheme; however, a very exible regularization may substitute the need for selection procedures. This paper investigates recently suggested adaptive regularization schemes. Some methods focus directly on minimizing an estimate of the generalization error (either algebraic or empirical) [1], [3], [5], [6], [7], [12], [13], whereas others starts from di erent criteria, e.g., the Bayesian evidence [2, Ch. 10], [7], [15], [16]. The evidence expresses basically the probability of the model, which is conceptually di erent from generalization error; however, asymptotically for large training data sets they will converge [15]. The papers is organized as follows: rst the basic model de nition, training and generalization is presented. Next, di erent adaptive regularization schemes are reviewed and extended. Finally, the experimental section presents a comparative study concerning linear models and feed-forward neural networks models for regression/time-series problems.